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ABSTRACT. We give an explicit description of hyperbolic Reinhardt domains DcC 2 
such that: (i) D has C fc -smooth boundary for some k > 1, (ii) D intersects at least 
one of the coordinate complex lines {zi = 0}, {22 = 0}, and (iii) D has noncompact 
automorphism group. We also give an example that explains why such a setting is 
natural for the case of hyperbolic domains and an example that indicates that the 
situation in C n for n > 3 is essentially more complicated than that in C 2 . 



0. Introduction and Results 

Let D be a Kobayashi- hyperbolic domain in C n , n > 2 (see [Ko] for terminology). 
Denote by Aut(-D) the group of holomorphic automorphisms of D. The group 
Aut(-D) with the topology of uniform convergence on compact subsets of D (the 
compact-open topology) is in fact a Lie group (see [Ko]). 

The present paper is motivated by results characterizing a domain by its auto- 
morphism group (see e.g. [R], [W], [BP1], [BP2]). More precisely, we assume that 
Aut(-D) is noncompact in the compact-open topology. Most of the known results 
deal with the case of bounded domains (see, however, [B]). In the present paper we 
consider possibly unbounded hyperbolic domains. Our thesis is that (unbounded) 
hyperbolic domains have some of the geometric characteristics of bounded domains. 
In particular, they are tractable for our studies. But they also exhibit new automor- 
phism group action phenomena, and are therefore of special interest. We present 
some of these new features in this work. 

Here we assume that D is a Reinhardt domain, i.e. a domain which the standard 
action of the n-dimensional torus T n on C™, 

zj i-> e^'Zj, <pj £ R, j = 1, . . . , n, (1) 



leaves invariant. In [FIK] we gave a complete classification of smoothly bounded 
Reinhardt domains with noncompact automorphism group, and in [IK] we extended 
this result to Reinhardt domains with boundary of any finite smoothness C k , k > 1. 
One of the main steps for obtaining these classifications was to show that the non- 
compactness of Aut(-D) is equivalent to that of Auto(D), the connected component 
of the identity in Aut(D). We will now explain this point in more detail, as it will 
provide some motivation for the results of the present paper. 
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Following [Sh], we denote by Aut a i g ((C*) ra ) the group of algebraic automorphisms 
of (C*) n , i.e. the group of mappings of the form 

Zi ^ XiZ^ 1 ...z%", i = l,...,n, (2) 

where Ai € C* = C \ {0}, € Z, and det(ajj) = ±1. For a hyperbolic Reinhardt 
domain D C C n , denote by Aut a i g (D) the subgroup of Aut(D) that consists of 
algebraic automorphisms of D, i.e. automorphisms induced by mappings from 
Aut a i g ((C*) n ). It is shown in [Kr] that Aut(L>) = Aut (£>) • Aut a i g (L>), where the 
" • " denotes the composition operation in Aut(D). Therefore if one can show 
that, for a hyperbolic Reinhardt domain D, Aut a i g (-D) is finite up to the action of 
T n (see (1)), then the noncompactness of Aut(D) is equivalent to that of Auto(-D) 
(see Proposition 1.1 in [FIK] for the case of bounded domains). Next, as is shown 
in [Kr], Auto(-D) admits an explicit description if D is mapped into its normalized 
form by a mapping of the form (2). This normalized form was the main tool that 
we used in [FIK], [IK]. 

Unfortunately, as the following example shows, for the case of hyperbolic Rein- 
hardt domains the group Aut a i g (D) may be essentially infinite, and therefore the 
scheme used in [FIK], [IK], fails. 

Example 1. Consider the Reinhardt domain DcC 2 

D = | sin ^log < log \z x z 2 \ < sin ^log j^-j^ + i j . (3) 

The boundary of D is clearly C°°-smooth. The group Aut a i g (D) is not finite up to 
the action of T 2 , since it contains all the mappings 

Zi i-> e nk z 1 , 
z 2 i-> e~ nk z 2 , 

for k € Z. This also shows, of course, that Aut(-D) is noncompact. 

To see that D is hyperbolic, consider the mapping /: D — > C, f(zi,z 2 ) = z\z 2 . 

It is easy to see that / maps D onto the annulus A = | e_1 < \ z \ < e ^ | which is a 

hyperbolic domain in C. The domains 

A 1 = |e"3 < \z\ < e^j , 

A 2 = |e _1 < \z\ < e~sj , 



A 



3 = |e4 < \z\ < e2 1 



obviously cover A, and each of the preimages Dj = f~ 1 (Aj), j = 1,2,3, is hyper- 
bolic since Dj is contained in a union of bounded pairwise nonintersecting domains. 
It then follows (see [PS]) that D is hyperbolic. □ 

It should be noted here that the domain (3) does not intersect the coordinate 
complex lines {z\ = 0}, {z 2 = 0} (note that, for any hyperbolic Reinhardt domain 
in C n not intersecting the coordinate hyperplanes, one has Aut(D) = Aut a i g (D) 
[Kr]). As the following proposition shows, in complex dimension n = 2, the sort of 
pathology described in Example 1 above cannot occur if the domain intersects at 
least one of the coordinate complex lines. 
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Proposition A. Let D C C 2 be a hyperbolic Reinhardt domain with C 1 -smooth 
boundary, and let D intersect at least one of the coordinate complex lines {zj = 0}, 
j = 1,2. Then Aut a i g (D) is finite up to the action ofT 2 . 

In particular, for such a domain D, Aut(D) is noncompact if and only if Aut®(D) 
is noncompact. 

The above proposition allows us to use the description of Auto(-D) from [Kr] to 
obtain the following classification result. 

Theorem. Let D C C 2 be a hyperbolic Reinhardt domain with C k -smooth bound- 
ary, k > 1, and let D intersect at least one of the coordinate complex lines {zj = 0}, 
j = 1,2. Assume also that Aut(D) is noncompact. Then D is biholomorphically 
equivalent to one of the following domains: 



{|zi| 2 + H« < l}, 



where either a < 0, or a = for some m G N, or a ^ for any 

2m 2m 

m G N and < a < — ; 

(a) {\ Zl \< i, (i - \ Zl \ 2 r < N < R(i - \zi\ 2 Y) , 

where 1 < R < oo and a < 0; 
(Hi) {e' 312112 < \z 2 \ < i?e /3|zi|2 J , where 1< R < oo, j3 G M, + 0. 

If k < oo and dD is not C°° -smooth, then D is biholomorphically equivalent to 
a domain of the form (i) with a / ^ for any m G N and < a < ^ • 

In case (i) the equivalence is given by dilations and a permutation of the coordi- 
nates; in cases (ii) and (Hi) the equivalence is given by a mapping of the form 

z a{1) i-> AV(i)^ (2)1 

^(2)^^(2)' 

where A,/i G C*, a G Z and a, a are permutations o/{l,2}. 

In the following example we show that, in complex dimension n > 3, an explicit 
classification result analogous to the above theorem does not exist if we do not 
impose extra conditions on the domain D. 

Example 2. Consider the domain DcC 3 given by 

D=\z: <t>{z) ee \ Zl \ 2 + (1 - |^| 2 ) 2 |z 2 | 2 p(|z 2 | 2 (l - N 2 ), |z 3 | 2 (l - N 2 )) 

1 v (4) 

+ (l-|zi| 2 ) 2 |z 3 | 2 -l<oL 



where p(xi,x 2 ) is a C°°-smooth function on R 2 such that p(x 1 ,x 2 ) > c > every- 
where, and the partial derivatives of p are nonnegative for xi,x 2 > 0. 
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To show that dD is smooth, we calculate 



d( t>_^(, n l- |2^oi„ |2„ , /i l„ |2m„ |4 d P 



+ (1 - \zi\ )\z 2 \ \z 3 \ h 2 z 3 ' 

^ n I |2\2— /" , /-, I |2m |2 dp 

- (1 - \zi\ ) z 2 p+ (1 - |zi| )\z 2 ] 



)) 



(5) 



OZ 2 \ OX\ J 

d ^ n i i2n2 f n i ,2m i2 d P , A 



It follows from (5) that not all the partial derivatives of can vanish simultaneously 

I*, 

d<f> 



at a point of dD. Indeed, if -i^-(p) = at some point p G dD then, at p, either 



\zi\ = 1 or zs = 0. If \z\\ = 1, then clearly g^(p) ^ 0. If \z\ \ ^ 1, z% = 0, and, in 

addition, §^{p) = 0, then z 2 = 0, and therefore \z\\ = 1, which is a contradiction. 
Therefore, is C°°-smooth. 

To show that D is hyperbolic, consider the holomorphic mapping f(zi, z 2 , z$) = 
z\ from D into C. Clearly, / maps D onto the unit disc A = \\z\ < 1}, which 
is a hyperbolic domain in C. Further, the discs A r = {\z\ < r} for r < 1 form a 
cover of A, and / _1 (A r ) is a bounded open subset of D for any such r. Thus, as 
in Example 1 above, we see that D is hyperbolic (see [PS]). 

Further, Aut(D) is noncompact since it contains the automorphisms 



/ z-L-a (l-az 1 )z 2 (l-azi)z 3 \ , . 



for \a\ < 1. □ 

Examples similar to Example 2 can be constructed in any complex dimension 
n > 3. They indicate that, most probably, there is no reasonable classification of 
smooth hyperbolic Reinhardt domains with noncompact automorphism group for 
n > 3 even in the case when the domains intersect the coordinate hyperplanes. 
Indeed, in Example 2 we have substantial freedom in choosing the function p. 

We note that the boundary of domain (4) contains a complex hyperplane z\ = a 
for any \a\ = 1. It may happen that, by imposing the extra condition of the 
finiteness of type in the sense of D'Angelo [D'A] on the boundary of the domain, 
one would eliminate the difficulty arising in Example 2 and obtain an explicit 
classification. It also should be observed that any point of the boundary of domain 
(4) with \zi \ = 1, z 2 = Z3 = is an orbit accumulation point for Aut(D) (see (6)); 
therefore, it is plausible that one needs the finite type condition only at such points 
(cf. the Greene/Krantz conjecture for bounded domains [GK]). 

This work was initiated while the first author was an Alexander von Humboldt 
Fellow at the University of Wuppertal. Research by the second author was sup- 
ported in part by NSF Grant DMS-9531967 and at MSRI by NSF Grant DMS- 
9022140. 
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Proof of Proposition A 

We consider two cases. 

Case 1. Suppose first that D intersects each of the coordinate complex lines 
{zj = 0}, j = 1, 2. Then any element F G Aut a i g (D) has the form 

zi >-> Xz a{1) 

Z 2 l-> ^(2) 

where A, ^ G C* and <r is a permutation of the set {1, 2}. 

First let a = id. We assume that mapping (7) is not of the form (1); hence either 
|A| ^ 1, or 1^1 ^ 1. By passing to the inverse mapping if necessary, we can also 
assume that |A| < 1, or \n\ < 1. 

Let |A| < 1. Take a point p G D of the form p = (c, 0) and apply the k th 
iteration F k of F to it: F k (p) = (X k c,0). Since |A| < 1, it follows that X k c -» 
as k — > oo, and therefore (0, 0) G L>. Since 3D is C 1 -smooth, we actually obtain 
that (0,0) G D. Therefore, for some e > 0, the disc A £ = {\z\\ < e,Z2 = 0} lies 
in D. By applying the k th iteration of F _1 to A e and letting k — > oo, we obtain 
(since |A _fc | — > oo) that the domain D contains the entire complex line {z 2 = 0} 
and therefore cannot be hyperbolic. The case of < 1 is treated similarly. Hence, 
|A| = = 1, and F is of the form (1). 

Suppose now that <t(1) = 2, <r(2) = 1. We will show that there exists no more 
than one automorphism of the form (7) with this a (up to mappings of the form 
(1)). Let F 1: F 2 be two such automorphisms, with Fj for j = 1, 2 given by 

Zi h-> Xjz 2 , 

Z 2 fljZi, 

where Xj,fij G C*. Then, for the composition F\ o F 2 _1 , we find that 

Ai 

Zi I ^ — Zi, 

Mi 

z 2 i-> — z 2 . 

Hence, by the preceding argument, |Ai| = \X 2 \ and = \fi 2 \; therefore F\ differs 
from F 2 by a mapping of the form (1). 

Case 2. Let D intersect only one of the coordinate complex lines, say {z\ = 0}. 
Then any element of Aut a i g (D) either has the form 

Z\ I— > A^lZn, 

(8) 

z 2 l-> /iZ 2 , 

or the form 

Z\ I— > XZlZn, 

(9) 

where A, /U G C*, a G Z. We will show that there is at most one element of Aut a i g (-D) 
of each of the forms (8) and (9). 
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Let Fj, j = 1, 2, be two automorphisms of the form (8) given by 

\ a i 
Z 2 !-»■ ^2, 

where A^, fij G C*, Oj £ Z. Then for F = F 1 o F 2 -1 we see that 

^^2 2 ~ ai ^2 ai ~ Q2 , 
^2 

Ml 

Z 2 ^2- 

M2 

Let L> = D n {zi = 0}. Then, since <9L> is C^-smooth, dist(A), {-z 2 = 0}) > 0. 
Clearly, F preserves Dq. Suppose now that < |/i 2 |- Then, by considering 
the images of -Do under iterations of F, we see that dist(Do, {z 2 = 0}) = which 
contradicts the smoothness of dD. Similarly, if > |/x 2 |, then by applying 
iterations of F _1 to Dq we obtain the same contradiction. Therefore, \ = \[i 2 \. 

By composing F 2 with a mapping of the form (1), we can now assume that 
Mi = M2 = M an d therefore F is given by 

21 ^i/i a2 - ai z 1 4 1 - a2 , 
A2 

Z 2 l-> Z 2 . 

The /c th iteration of -F then has the form 

Z 2 >->■ Z 2 . 



We now observe that there exist e > and a disc A C C such that the bidisc 
{\zi\ < e,z 2 G A} lies in D. Let A ejC = {|^i| < e,z 2 = c} for c G A. If for 

~ x (01-02) 

some c G A we have ^ • ^ > 1 then, by applying the iterations F fc to 

A 6jC and letting k — > 00, we see that the complex line {^2 = c} belongs entirely 
to D; this conclusion contradicts the hyperbolicity of D. Similarly, if for some 



c G A, 



Ai 






A 2 







(01-12) 



contradiction. Therefore, 



< 1 , then applying iterations of F 

(oi-o 2 ) 



Ai 




c 


A 2 




/< 



to A £jC yields the same 
0*2, |Ai| = |A 2 |. 



= 1 in A, and hence a\ 

Thus Fi differs from F 2 by a mapping of the form (1). 

The case of mappings of the form (9) is treated analogously. This completes the 
proof of the proposition. □ 



2. Proof of Theorem 

We will use the following description of Auto(-D) from [Kr]. Any hyperbolic 
Reinhardt domain in C n can — by a biholomorphic mapping of the form (2) — be 
put into a normalized form G written as follows. There exist integers < s < 
t < p < n and rij > 1, i = l,...,p, with Yli=i n * = n -> an d re al numbers oct, 
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i = 1, . . . ,s, j = t+1, ... ,p, such that if we set z l = (z ni+ ... +ni _ 1+1 , . . . , z ni+ ... +ni ) , 
i = 1, ... ,p, then G := Gf]{z i = 0, i = 1, . . . , *} is a hyperbolic Reinhardt domain 
in C n * +1 x • • • x C np , and G can be written in the form 



G= { Iz 1 ! < l,...,|z s | < 1, 



z t+i 



v* + ! 



(10) 



nu (i-i^i 2 ) a 'nUe X p(^i^i 2 )) GG }' 



for some real numbers j = s + 1, . . . , t, k = t + 1, . . . ,p. A normalized form can 
be chosen so that Auto(G) is given by the following formulas: 



i A i z i + b i 

Z I ^ : : rr . % — 1 ..... S. 

c l z l + <P 

z 3 B 3 z 3 + e 3 , j = s + 1, . . . , t, 



C ^ " % — , k = t + l,...,p, 

n: =1 (c^ i +d i ) 2a ' 



1 e SU(rii, 1), i = 1, . . . ,s, 



where 



B 3 e U( nj ), e 3 € C n > , j = s + 1, . . . , t, 
C k eU{n k ), fe = t + l,...,p. 

The above classification implies that Auto(G) is noncompact only if t > 0. 

Now let n = 2. Clearly there are the following possibilities for a hyperbolic 
Reinhardt domain GcC (see (10)): 



(i) G = {\z 2 \<R}, 0<i?<oo; 

(ii) G = {r < \z 2 \ < R}, 0<r<R<oo; 

(iii) G = {0 < \z 2 \ < R}, 0<R<oo. 



This observation allows us to list all normalized forms of hyperbolic Reinhardt 
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domains in C 2 as follows 

G={\z 1 \<l,\z 2 \<R(l-\z 1 \ 2 ) a }, 

0<R<oo, ael, (11) 
G = {\zt\ < l,r(l - \ Zl \ 2 ) a < \z 2 \ < R(l - \ Zl \ 2 ) a } , 

<r < R<co, aeK, (12) 

g = {\z!\ < 1,0 < \ Z2 \ < r(i - M 2 r} . 

0<i?<oc, ael, (13) 
G={re^ 2 <\z 2 \<Re^ 2 }, 

< r <R<oo, (3 € R, 0^0, (14) 
G= |o< 1 <i?e /3 l Zl l 2 }, 

0<i?<oo, /3€R, /3^0. (15) 

We are now going to select only those among the normalized forms (11)— (15) that 
can be the images of domains with C fc -smooth boundary under normalizing map- 
pings of the form (2). We will treat each of cases (11)— (15) separately 

Domain of type (11). Observe first that, since the domain G contains the 
origin, the normalizing mapping is linear (actually, it is given by dilations and a 
permutation of the coordinates). Therefore, the domain G is a normalized form of 
a Reinhardt domain with C fc -smooth boundary iff dG is also C fc -smooth. Hence 
a/0 (for otherwise G is a bidisc). If a > 0, then G has a C fc -smooth boundary iff 
either a = for some m G N, or a / ^ for any m € N and a < j^. If a < 0, 
then dG is C°°-smooth. It is also clear that, for k < oo, dG has C fc -smooth, but 
not C°°-smooth, boundary iff a ^ for any m € N, and < a < ik-. 

Domain of type (12). First of all, if a < 0, then dG is C°°-smooth. It is also 
clear that, if a = 0, then G cannot be a normalized form of any Reinhardt domain 
with everywhere C fc -smooth boundary for k > 1. 

Assume now that a > 0, and suppose first that R < oo. Then dG is C°°- 
smooth everywhere except at the points where \z\\ = 1, z 2 = 0. By applying the 
transformation 

Zi >-> 2l, 

1 (16) 
z 2 i-> — , 

z 2 

we produce the following domain with C^-smooth boundary 

{n < i, ^(i - Nil 2 )"" < 1*2 1 < J(i - MT Q } • 

Let a > and i? = oo. We claim that in this case G cannot be a normalized form 
of a Reinhardt domain with everywhere C fc -smooth boundary for k > 1. Indeed, 
this is easy to see if we notice that the general form (up to permutation of the 
components) of any mapping (2) that is biholomorphic on G is as follows 

z\ \— > Xziz 2 , 

±i (17) 

Z 2 H-> /XZ 2 , 
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where A,/i6C*,o6 

It is also easy to see that, for k < oo, no domain (12) can be a normalized form 
of a Reinhardt domain with C fc -smooth, but not C°°-smooth, boundary. 

Domain of type (13). By transformation (16), G is mapped into a domain of 
the form (12) corresponding to the case R = oo, so it can be treated as above. 

Domain of type (14). The boundary dG of G is C°°-smooth. Also, if k < oo, 
then G cannot be a normalized form of any Reinhardt domain with C fc -smooth, 
but not C°°-smooth, boundary; this assertion is proved by the same argument as 
we used for domains of the form (12) above (see (17)). 

Domain of type (15). By transformation (16), the domain G is mapped into 
a domain of the form (14) corresponding to the case R = oo, so it can be treated 
as above. 

The theorem is proved. □ 
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